Skip to content

Conversation

@bkhouri
Copy link
Contributor

@bkhouri bkhouri commented Dec 1, 2025

This proposal adds test metadata (tags, bugs and timeLimit) to Swift Testing's event JSON ABI, enabling richer external tooling capabilities.

Links: Implementation | Forum Pitch

@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch from ad4afed to bf363fb Compare December 1, 2025 21:33
@stmontgomery stmontgomery self-requested a review December 1, 2025 23:10
@stmontgomery stmontgomery added workgroup: needs pitch This proposal needs to spend more time in pitch before it can be reviewed workgroup: needs development This proposal needs to be developed further before it can be reviewed TWG Contains topics under the domain of the Testing Workgroup labels Dec 1, 2025
@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch 2 times, most recently from 8a7e8b1 to e7325c8 Compare December 9, 2025 19:34
@bkhouri bkhouri changed the title SwiftTesting: Enhance Event Stream JSON ABI SwiftTesting: Include metadata for tags, bug, and time limit traits in event stream Dec 10, 2025
@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch 2 times, most recently from 55d172e to f56f077 Compare December 10, 2025 04:07
+ "title": <string> ; the human readable bug title
+} ;
+
+<characters> ::= "" | <characters> <character>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to spell these out. Please remove them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK


<test-id> ::= <string> ; an opaque string representing the test case
+
+<tag> ::= "." <chacters> | <tag> ; a string representation of a tag
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is invalid syntax. What are you trying to describe here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to describe that a tag has this syntax .<charaters>[.<characters|.....]

ie: .Foo.bar.bar

But this conversation might be moot based on the output of https://github.com/swiftlang/swift-evolution/pull/3040/files#r2607028858

}
],
"tags": [
".blue",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should think about whether the leading period should be encoded in the JSON. Since this schema may be used by other libraries that don't use Swift's dot syntax, my gut says they shouldn't be encoded and we should just define <tag> ::= <string>.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should think about whether the leading period should be encoded in the JSON

I don't have a strong opinion here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree w/ @grynspan and think we shouldn't necessarily specify in the schema itself that there will always be a leading period character. But I do think the proposal should clearly specify whether, in the encoded string, the leading period and any subsequent punctuation will be preserved as it was written in source. The main thing I'm trying to convey is that we ought to clearly define the behavior, whatever it is.

And on this specific point, I would suggest you define the behavior such that it will precisely preserve the way the tag was spelled in source. So if the code had .tags(.blue) the string would be ".blue". If it had .tags(Tag.red), the string would be "Tag.red", and so on. The latter example differs from the example you show in the current proposal, so that represents a change.

Copy link
Contributor

@grynspan grynspan Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be the wrong abstraction though, as it would mean Tag.red and .red don't compare equal. (I don't think that string is available at runtime anyway!) Furthermore, preserving the source would prevent the use of this feature with libraries that define tags using different syntax. For example, C++ might have something using ::, or Objective-C might just use @"red".

I'd like to just define it formally as <string> and document that Swift Testing specifically normalizes to "red" since that would be most compatible with what other libraries or tagging syntaxes would reasonably do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to that. To be comprehensive about this, how would examples which include interstitial punctuation, like .tags(.NestedType.bar, .NestedType.AnotherNestedType.baz) be encoded in the JSON? As "NestedType.bar" and "NestedType.AnotherNestedType.baz", respectively, or something else?

In effect, this will mean we'll be treating the optional (Testing.)Tag type namespace prefixes as "special", and stripping them when present, along with the leading period. Which is fine, but it should be spelled out in the proposal. (Maybe it would be helpful to include a Markdown table with several of these examples.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @grynspan that Tag.red and .red should be represented the same way in the event stream. Do we want to consider having the JSON represents each tags as an array, or maybe an object, instead of string? This might allow some IDE's to represent a tree structure of all tags?

Copy link
Contributor

@stmontgomery stmontgomery Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in favor of the proposal's current behavior for the two examples it shows—that .tags(.blue) and .tags(Tag.red) are represented as ".blue" and ".red", respectively.

Can we include one more example, just to clarify the handling of nested tags? Specifically, can we document that a two-level tag like .tags(.Foo.bar) would map to the string ".Foo.bar"?

Copy link
Contributor

@stmontgomery stmontgomery Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, hold on. More investigation has revealed an interesting data point: the sourcekit-lsp indexer currently does not include the leading period, so that suggests omitting it in the JSON event stream representation would make things align better and ultimately, that's one of the biggest considerations.

So here's my current preference:

Trait String
.tags(.blue) "blue"
.tags(Tag.red) "red"
.tags(.Foo.bar) "Foo.bar"
.tags(Tag.Foo.bar) "Foo.bar"
.tags(Testing.Tag.Foo.bar) "Foo.bar"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to use a string with the leading . removed. Do we know where sourcekit-lsp indexer gets the value from? Does it generate that string itself? Is there value in creating a Swift Testing API that would be the source of truth, which sourcekit-lsp can then eventually start using?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's conceivable, but pretty far outside the scope of this proposal so I'd recommend breaking that out into a separate discussion

@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch from f56f077 to 0a650cd Compare December 10, 2025 19:02
@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch 2 times, most recently from 7530592 to 113f1a1 Compare December 10, 2025 19:04
@bkhouri bkhouri force-pushed the t/main/augment_event_stream_json branch from 113f1a1 to 71d2c9b Compare December 18, 2025 04:02
.tags(Tag.red),
.bug("https://my.defect.com/1234"),
.bug("other defect"),
.timeLimit(Swift.Duration.seconds(testTimeLimit + 100)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI these three aren't valid. Use .minutes() instead (the overloads that take Swift.Duration are unavailable.)


```json
{
"kind": "test",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: missing the "version" field.

Comment on lines +101 to +103
public static var blue: Self {
Tag(kind: .staticMember("blue"))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static var blue: Self {
Tag(kind: .staticMember("blue"))
}
@Tag public static var blue: Self

Comment on lines +106 to +108
public static var red: Self {
Tag(kind: .staticMember("red"))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static var red: Self {
Tag(kind: .staticMember("red"))
}
@Tag public static var red: Self

.tags(.blue),
.tags(Tag.red),
.bug("https://my.defect.com/1234"),
.bug("other defect"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.bug("other defect"),
.bug(id: "12345", "other defect"),

The compiler will reject an invalid URL string here rather than interpreting it as a comment.

"url": "https:\/\/my.defect.com\/1234"
},
{
"url": "other defect"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"url": "other defect"
"id": "12345"
"title": "other defect"

<...SNIP...>,
"bugs": [
{
"url": "https:\/\/my.defect.com\/1234"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"url": "https:\/\/my.defect.com\/1234"
"url": "https://my.defect.com/1234"

If we're escaping forward slashes right now, that's a bug on our side I think.

- **`timeLimitInSeconds` vs `timeLimit`**: We chose the shorter `timeLimit` name for
consistency with Swift Testing's existing API, with the time unit documented in the
schema specification. The naming convention was discussed with the Testing Workgroup
and it was decided that a seperata proposal should be made on how to represent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
and it was decided that a seperata proposal should be made on how to represent
and it was decided that a seperate proposal should be made on how to represent


## Future directions

This enhancement establishes future richer tooling experiences:
Copy link
Contributor

@grynspan grynspan Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This enhancement establishes future richer tooling experiences:

I don't think this sentence is meaningful.


No existing functionality is affected, making this a purely additive enhancement.

## Integration with supporting tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest reworking this section. The leading sentence doesn't say anything and then bullet points are vague and promise functionality that's beyond the scope of this proposal. Perhaps something like:

"Supporting tools like IDEs and CI systems consume the JSON event stream and will be able to use this new information to provide richer developer experiences. For example, an IDE could allow developers to sort the content of a test report by tag, or could highlight failing tests with associated bugs that are incorrectly marked fixed."

(But feel free to rewrite that of course, that's just off the top of my head.)

- **Performance Monitoring**: Can track and alert on time limit violations
- **Bug Tracking Integration**: Can correlate test results with known issues

### Migration Path
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably drop this section as the "migration path" isn't very long. Or just rewrite it to something like "tools that request this version of the JSON event stream schema will automatically have access to the new fields."


## Alternatives considered

### Alternative Data Structures
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Alternative Data Structures

Comment on lines +197 to +199
### Unconditionally include optional field
- We selected conditional inclusion to keep JSON output clean and avoid null values,
improving the developer experience for tools consuming the data.
Copy link
Contributor

@grynspan grynspan Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Unconditionally include optional field
- We selected conditional inclusion to keep JSON output clean and avoid null values,
improving the developer experience for tools consuming the data.
- **Unconditionally include optional fields**: We selected conditional inclusion
to keep JSON output clean and avoid null values, improving the developer
experience for tools consuming the data.

and it was decided that a seperata proposal should be made on how to represent
the time units in the name/value.

### Potential Extensions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Potential Extensions


This enhancement establishes future richer tooling experiences:

### Alternative Field Naming
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Alternative Field Naming

Comment on lines +150 to +160
This proposal maintains full backward compatibility through careful design:

- **ABI Version Protection**: New fields are conditionally included based on ABI
version checks, ensuring older tools continue to function without modification
- **Experimental Feature Migration**: The existing experimental `_tags` field is
replaced with the `tags` array. Since experimental features don't provide
stability guarantees, this replacement doesn't constitute a breaking change
- **Graceful Degradation**: Tools that don't expect the new fields will simply ignore
them, while updated tools can leverage the enhanced metadata

No existing functionality is affected, making this a purely additive enhancement.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This proposal maintains full backward compatibility through careful design:
- **ABI Version Protection**: New fields are conditionally included based on ABI
version checks, ensuring older tools continue to function without modification
- **Experimental Feature Migration**: The existing experimental `_tags` field is
replaced with the `tags` array. Since experimental features don't provide
stability guarantees, this replacement doesn't constitute a breaking change
- **Graceful Degradation**: Tools that don't expect the new fields will simply ignore
them, while updated tools can leverage the enhanced metadata
No existing functionality is affected, making this a purely additive enhancement.
This proposal is additive only. Tools using earlier versions of the JSON event stream schema are not affected.

+ ["url": <string>,] ; the bug URL
+ ["id": <string>,] ; the bug id
+ ["title": <string>] ; the human readable bug title
+} ;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
+} ;
+}


### Sample JSON Output

Given the following Test Case
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Given the following Test Case
Given the following test case:

Comment on lines +58 to +62

### Implementation Strategy

Fields are only included when the test actually has at least one matching trait applied, preserving
backwards compatibility with previous versions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Implementation Strategy
Fields are only included when the test actually has at least one matching trait applied, preserving
backwards compatibility with previous versions.

Comment on lines +55 to +57
This enhancement builds upon the existing test metadata infrastructure already used
internally by Swift Testing. The implementation reuses established data structures,
ensuring consistency and minimizing complexity.
Copy link
Contributor

@grynspan grynspan Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This enhancement builds upon the existing test metadata infrastructure already used
internally by Swift Testing. The implementation reuses established data structures,
ensuring consistency and minimizing complexity.
We propose adding `"tags"`, `"bugs"`, and `"timeLimit"` fields to the existing `<test-suite>` and `<test-function>` structures in Swift Testing's JSON event stream schema starting with version `"6.3"`.

Comment on lines +49 to +51

These additions leverage existing internal data structures, ensuring minimal performance
impact while maximizing the value delivered to external tools.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
These additions leverage existing internal data structures, ensuring minimal performance
impact while maximizing the value delivered to external tools.

Comment on lines +39 to +48

We propose enriching the test payload in the event JSON stream by adding three
metadata fields:

- **`tags`**: An array of strings where each item represents a single tag applied to the test,
enabling categorization and filtering
- **`bugs`**: An array of bug references, providing traceability between tests
and issue tracking
- **`timeLimit`**: The test's time limit in seconds, enabling performance monitoring
and timeout analysis
Copy link
Contributor

@grynspan grynspan Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We propose enriching the test payload in the event JSON stream by adding three
metadata fields:
- **`tags`**: An array of strings where each item represents a single tag applied to the test,
enabling categorization and filtering
- **`bugs`**: An array of bug references, providing traceability between tests
and issue tracking
- **`timeLimit`**: The test's time limit in seconds, enabling performance monitoring
and timeout analysis
We propose adding new fields to Swift Testing's JSON event stream schema that represent the described traits.

Comment on lines +34 to +36

By exposing this information, we unlock new possibilities for Swift Testing
tooling ecosystem.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
By exposing this information, we unlock new possibilities for Swift Testing
tooling ecosystem.

This is the problem statement, not the solution.


Currently missing from the JSON output are:
- **Test tags**: Used for categorization
- **Bug associations**: Critical for tracking which tests verify specific bug fixes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **Bug associations**: Critical for tracking which tests verify specific bug fixes
- **Bug associations**: Track bugs associated with specific tests

Currently missing from the JSON output are:
- **Test tags**: Used for categorization
- **Bug associations**: Critical for tracking which tests verify specific bug fixes
- **Time limits**: Essential for performance monitoring and timeout management
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **Time limits**: Essential for performance monitoring and timeout management
- **Time limits**: Useful for performance monitoring and timeout management


## Motivation

Swift Testing's event JSON stream provides data for external tooling,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Swift Testing's event JSON stream provides data for external tooling,
Swift Testing's JSON event stream provides data for external tooling,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

TWG Contains topics under the domain of the Testing Workgroup workgroup: needs development This proposal needs to be developed further before it can be reviewed workgroup: needs pitch This proposal needs to spend more time in pitch before it can be reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants